Incentivizing foodstuff consumption through the use of augmented reality features

ABSTRACT

Introduced here are techniques for incentivizing the consumption of foodstuffs through the use of augmented reality features. A parent may initiate an application on a mobile phone and scan a plate of one or more foodstuffs by initiating a live view captured by a camera. The parent can then select a given foodstuff that a child is unlikely to consume. When the child views the given foodstuff through the mobile phone, a portion of an augmented reality feature could be shown. For example, the child may be able to see the tail of a creature, a corner of a item, etc. As the child consumes the given foodstuff, additional portion(s) of the augmented reality feature are exposed, and the augmented reality feature becomes increasingly visible. Such a technique incentivizes the child to continue eating the given foodstuff so that the augmented reality feature can be fully seen.

CROSS REFERENCE TO RELATED APPLICATION

This patent application is a continuation of U.S. patent applicationSer. No. 15/672,142, filed Aug. 8, 2017, which is a continuation of U.S.patent application Ser. No. 15/596,643, filed May 16, 2017, which areincorporated by reference herein in their entirety.

RELATED FIELD

Various embodiments pertain to computer-generated sensory input and,more specifically, to augmented reality features that can be used toincentivize the consumption of foodstuffs.

BACKGROUND

Recent studies on children's eating habits have shown that many Americantoddlers go without vegetables on any given day. For example, one infour 6-month-olds to 11-month-olds and one in five 1-year-olds reportedno vegetable consumption at all on days on which they were surveyed.Studies have also shown a general decline in the consumption of healthyfruits and vegetables among toddlers, and that most children areconsuming less than half the recommended amount of fruits andvegetables. Because children's eating patterns develop at a young age,poor eating habits when young can put children on a path of poornutrition that often extends into adulthood.

However, fruits and vegetables are high in the vitamins, minerals,antioxidants, and dietary fibers necessary to promote goodhealth/development and protect against disease/illness. Fruits andvegetables have been shown to improve overall nutrition, weightmanagement, intestinal health, academic performance, disease prevention,etc. Consequently, the American Academy of Pediatrics (AAP) recommendsthat vegetables be consumed with every meal and snack. But it can bedifficult to get children to consistently consume nutritious foods.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of various systems and techniques for creating augmentedreality features (also referred to as “the technology”) are illustratedby way of example and not limitation in the accompanying drawings, inwhich like references indicate similar elements. Various objects,features, and characteristics of the technology will become moreapparent to those skilled in the art from a study of the DetailedDescription in conjunction with the accompanying drawings.

FIG. 1A illustrates how the application may be configured to initiate alive view of a branded foodstuff.

FIG. 1B illustrates how the application may be used to scan a brandedcode affixed to the packaging of a branded foodstuff.

FIG. 2A depicts a live view of an unbranded foodstuff.

FIG. 2B depicts a live view of multiple unbranded foodstuffs on a plate.

FIG. 3 illustrates how a portion of an augmented reality feature may bevisible when a child views a foodstuff through a mobile phone.

FIG. 4A depicts a bowl after a child has consumed the foodstuff(s)retained within the bowl (here, cereal and milk).

FIG. 4B illustrates how collections of augmented reality features can becollected by the child.

FIG. 4C illustrates how collections of augmented reality features can becollected by the child.

FIG. 5 illustrates how augmented reality features (also referred to as“digital collectibles”) acquired by a child may be part of a collection.

FIG. 6 depicts an example of a network environment that includes amobile phone having an application configured to present augmentedreality features that are embedded within or disposed beneathfoodstuff(s) and a network-accessible server system responsible forsupporting the application.

FIG. 7 depicts an application that allows augmented reality features tobe created for presentation by a computing device.

FIG. 8 depicts a flow diagram of a process for incentivizing theconsumption of foodstuff through the use of augmented reality features.

FIG. 9 depicts a flow diagram of a process for creating augmentedreality features that can be used to incentivize the consumption offoodstuffs.

FIG. 10 is a block diagram illustrating an example of a processingsystem in which at least some operations described herein can beimplemented.

The figures depict various embodiments of the technology for the purposeof illustration only. Those skilled in the art will recognize thatalternative embodiments may be employed without departing from theprinciples of the technology.

DETAILED DESCRIPTION

Introduced here are various systems and techniques for incentivizing theconsumption of foodstuffs through the use of augmented reality features.Such technology can be used to address a fundamental challenge, namely,persuading children to eat nutritional foods, such as fruits andvegetables.

For example, a parent may initiate an application on a mobile phone, andthen scan a plate of one or more foodstuffs by initiating a live viewthat is captured by a camera. The term “foodstuffs” is used to generallyrefer to any type of food or beverage. While the application may beconfigured to automatically identify recognizable features indicative ofa certain foodstuff (e.g., shapes and colors), the parent will typicallyneed to indicate which foodstuff(s) on the plate will be associated withan augmented reality feature.

For example, the parent may select a given foodstuff (e.g., spinach)that a child is least likely to consume from multiple foodstuffs on aplate, cup, or bowl. When the child views the given foodstuff throughthe mobile phone, a portion of an augmented reality feature could beshown. For example, the child may be able to see the tail of a dragon, acorner of a treasure chest, etc. The portion that is shown can also bereferred to as a “hint” or “teaser” of the digitally-animated rewardthat awaits the child once the child finishes consuming some or all ofthe given foodstuff.

As the child consumes the given foodstuff, additional portion(s) of theaugmented reality feature are exposed, and the augmented reality featurebecomes increasingly visible. Such a technique incentivizes the child tocontinue eating the given foodstuff so that the augmented realityfeature can be fully seen. After the child has finished eating the givenfoodstuff, the parent or the child may once again scan the plate byinitiating the live view. The application may detect that some or all ofthe given foodstuff is no longer present (which is indicative ofconsumption by the child), and then present a “payoff.” In someembodiments the “payoff” is simply presenting the entire augmentedreality feature for viewing by the child, while in other embodiments the“payoff” is a digital collectible corresponding to the augmented realityfeature.

Some embodiments have been described in the context of parents andchildren for the purpose of illustration. Note, however, that thetechnology can be used by individuals of any age. For example, adultsmay also collect augmented reality features that are associated withtelevision programs, video games, etc. Although certain embodiments maybe described in the context of mobile phones, those skilled in the artwill recognize that such embodiments have been selected for the purposeof illustration only. The technology could be used in combination withany computing device that is able to present augmented reality content,including personal computers, tablet computers, personal digitalassistants (PDAs), game consoles (e.g., Sony PlayStation® or MicrosoftXbox®), music players (e.g., Apple iPod Touch®), wearable electronicdevices (e.g., watches or fitness bands), network-connected (“smart”)devices (e.g., televisions and home assistant devices),virtual/augmented reality systems (e.g., head-mounted displays such asOculus Rift® and Microsoft Hololens®), or other electronic devices.

For example, a child could wear a head-mounted display that allows thechild to have a live direct/indirect view of a physical, real-worldenvironment whose elements are augmented by computer-generated sensoryinput (e.g., one or more augmented reality features). The augmentedreality feature(s) may be created responsive to instructions generatedby the head-mounted display or another computing device to which thehead-mounted display is communicatively coupled. For example, anapplication residing on a mobile phone associated with a parent maygenerate instructions for creating the augmented reality feature(s) andtransmit those instructions to the head-mounted display. As anotherexample, a network-connected server system may generate instructions forcreating the augmented reality feature(s) and transmit thoseinstructions to the mobile phone or the head-mounted display.

These computing devices may communicate with one another via a wiredcommunication channel or a wireless communication channel. For example,the computing devices may communicate via a short range wirelesstechnology/protocol, such as Wi-Fi, Bluetooth®, Near Field Communication(NFC), cellular, infrared, radio-frequency identification (RFID), etc.

Oftentimes, the embodiments described herein will be tailored for one ormore particular environments. Accordingly, embodiments may be describedwith reference to certain devices (e.g., mobile phones) or networks(e.g., wireless local area networks (WLANs) configured for use inhomes). However, those skilled in the art will recognize that featuresof the technology are transferable across different devices,environments, network types, network topologies, etc.

Terminology

Brief definitions of terms, abbreviations, and phrases used throughoutthe specification are given below.

Reference in this specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the disclosure. The appearances of the phrase “in some embodiments”are not necessarily referring to the same embodiments, nor are theynecessarily referring to separate or alternative embodiments that aremutually exclusive of one another. Moreover, various features aredescribed that may be exhibited by some embodiments but not others.Similarly, various requirements are described that may be requirementsfor some embodiments but not others.

Unless the context clearly requires otherwise, the words “comprise” and“comprising” are to be construed in an inclusive sense rather than anexclusive or exhaustive sense (i.e., in the sense of “including, but notlimited to”). The terms “connected,” “coupled,” or any variant thereofincludes any connection or coupling between two or more elements, eitherdirect or indirect. The coupling or connection between the elements canbe physical, logical, or a combination thereof. For example, two devicesmay be coupled directly to one another or via one or more intermediarychannels/devices. Devices may also be coupled in such a way thatinformation can be passed there between, despite not sharing anyphysical connection with one another. The words “associate with,”meanwhile, mean connecting or relating objects, items, etc.

Where the context permits, words used in the singular sense or theplural sense may also be used in the plural sense or the singular sense,respectively. The word “or,” in reference to a list of two or moreitems, covers all of the following interpretations of the word: any ofthe items in the list, all of the items in the list, and any combinationof the items in the list. If the specification states a component orfeature “may,” “can,” “could,” or “might” be included or have acharacteristic, that particular component or feature is not required tobe included or have the characteristic in every embodiment.

The term “module” refers broadly to software, hardware, and/or firmwarecomponents. Modules are typically functional components that cangenerate useful data or other output using specified input(s). A modulemay or may not be self-contained. A software program or application mayinclude one or more modules.

The terminology used in the Detailed Description is intended to beinterpreted in its broadest reasonable manner, even though it is beingused in conjunction with certain embodiments. The terms used in thisspecification generally have their ordinary meanings in the art, in thecontext of the disclosure as a whole and in the specific context whereeach term is used. For convenience, certain terms may be highlightedusing, for example, capitalization, italics, and/or quotation marks.However, the use of highlighting has no influence on the scope andmeaning of a term. The scope and meaning of a term is the same, in thesame context, whether or not it is highlighted.

Consequently, although alternative language and synonyms may be used forsome terms, special significance is not to be placed upon whether or nota term is elaborated or discussed herein. A recital of one or moresynonyms does not exclude the use of other synonyms. The use of examplesanywhere in this specification, including examples of any termsdiscussed herein, is intended to be illustrative only. These examplesare not intended to limit the scope and meaning of the disclosure or ofany exemplified term. Likewise, the disclosure is not limited to thevarious embodiments described below.

System Topology Overview

Various systems and techniques for incentivizing the consumption offoodstuffs through the use of augmented reality features. Examples ofaugmented reality features include digital representations of creatures(e.g., humanoid characters and non-humanoid characters, such as dragonsand dinosaurs) and items (e.g., treasure chests, trading cards, jewels,coins). In some embodiments, the digital representations are createdusing real textures (e.g., two-dimensional (2D) or three-dimensional(3D) footage/photographs) that are mapped onto computer-generatedstructures. For example, real images (or video footage sequences) of ashark could be mapped onto a 3D animated sequence. In other embodiments,the digital represents are illustrations not based on real photos/video.The term “foodstuffs,” meanwhile, is used to generally refer to any typeof food or beverage.

Generally, a parent initiates an application on a mobile phone and scansone or more foodstuffs by initiating a live view captured by an opticalsensor (e.g., a camera) or capturing a static image using the opticalsensor. The parent may initiate the application by selecting acorresponding icon presented by the mobile phone, and then direct afocal point of the optical sensor toward the foodstuff.

Several of the embodiments described herein pertain to instances wherethe type of augmented reality feature that is shown to a child dependson a matching cup/bowl/plate. Said another way, augmented realityfeatures may uniquely correspond to certain foodstuffs.

FIG. 1A illustrates how the application may be configured to initiate alive view of a branded foodstuff. A “branded foodstuff” is a foodstuffthat is associated with a particular brand. In some embodiments, theapplication parses the live video feed captured by the optical sensorfor real-time recognition purposes in order to identify alphanumericcharacter(s), symbol(s), and/or design(s) that are indicative of aparticular brand. Together, these elements define a particular syntax orpattern that corresponds to the particular brand.

Here, for example, the application may identify the type of foodstuffbased on the presence of the terms “Hidden Monsters.” There may bepremium augmented reality features (also referred to as “premiumcollectibles” or “premium incentives”) that require specific brandedfoodstuffs be purchased from participating partners (e.g.,grocery/retail stores or foodstuff manufacturers).

In some embodiments, the specific foodstuffs come with branded codesthat can be scanned by the parent using the mobile phone. FIG. 1Billustrates how the application may be used to scan a branded codeaffixed to the packaging of a branded foodstuff. The branded code couldalso be affixed to a loose item (e.g., a card or a toy) that is securedwithin the packaging of the branded foodstuff. For example, a brandedcode could be affixed to a paper card that is placed with a cereal box.Additionally or alternatively, an individual (e.g., a parent or child)could scan a purchase receipt as alternate proof of foodstuff purchase.

A branded code can be composed of some combination of machine-readableelements, human-readable elements, and/or structural elements.Machine-readable elements (e.g., bar codes and Quick Response (QR)codes) are printed or electronically-displayed codes that are designedto be read and interpreted by a computing device. Machine-readableelements can include extractable information, such as foodstuff type,foodstuff manufacturer, etc. Human-readable elements (e.g., text andimages) may be co-located with the machine-readable elements andidentified using various optical character recognition (OCR) techniques.Structural elements (e.g., emblems, horizontal lines, and solid bars)may also be used to identify a branded foodstuff from a branded code.

FIG. 2A depicts a live view of an unbranded foodstuff, and FIG. 2Bdepicts a live view of multiple unbranded foodstuffs on a plate. An“unbranded foodstuff” is a foodstuff that is not associated with aparticular brand.

In some embodiments, the application is configured to automaticallyidentify recognizable foodstuff features. For example, the applicationmay apply image processing algorithms to identify certain shapes and/orcolors that are indicative of certain foodstuffs (e.g., green spheresfor peas, orange cylinders for carrots, white liquid for milk). Morespecifically, the application may perform image segmentation (e.g.,thresholding methods such as Otsu's method, or color-based segmentationsuch as K-means clustering) on individual frames of the live video feedto isolate regions and objects of interest.

In other embodiments, the application identifies foodstuff(s) uponreceiving user input from a parent who indicates which foodstuff(s)should be associated with an augmented reality feature. For example, ifthe parent wanted to associate the peas of FIG. 2B with an augmentedreality feature, then the parent may be able to simply select the moundof peas presented on the screen of the mobile phone (e.g., by tappingthe display of the mobile phone). Thus, the parent could select afoodstuff that a child is least likely to consume from one or morefoodstuffs on a plate.

The application need not necessarily know the type of the selectedfoodstuff. That is, the application need not necessarily know whetherthe selected foodstuff is peas, carrots, spinach, etc. Instead, theapplication may only care about the geometrical characteristics of theselected foodstuff. Knowledge of the geometrical characteristics allowsthe application to realistically embed augmented reality feature(s)within the selected foodstuff.

FIG. 3 illustrates how a portion of an augmented reality feature may bevisible when a child views a foodstuff through a mobile phone. Theportion of the augmented reality feature that is shown can also bereferred to as a “hint” or “teaser” of the reward that awaits the childonce she finishes consuming some or all of the foodstuff. While thefoodstuff shown here is associated with a single augmented realityfeature, a foodstuff could also be associated with multiple augmentedreality features. For example, multiple digitally-animated coins/jewelsmay be distributed throughout a bowl of cereal or a mound of peas.

The visible portion(s) of the augmented reality feature indicate thatsomething is hidden within or beneath the foodstuff. Here, for example,portions of a digitally-animated dragon are visible when the foodstuffis viewed through the mobile phone. As noted above, augmented realityfeatures can take many different forms, including creatures (e.g.,humanoid characters and non-humanoid characters, such as dragons anddinosaurs) and items (e.g., treasure chests, trading cards, jewels,coins).

Animations may also be used to indicate the realism of the augmentedreality features. In some embodiments, different appendages (e.g., arms,legs, tails, head) may emerge from a foodstuff over time. For example,the digitally-animated dragon of FIG. 3 may swim through the bowl ofcereal over time. Additionally or alternatively, the foodstuff itselfmay appear as though it is animated. For example, the surface of thebowl of milk may include ripples caused by certain swimming actions orbubbles caused by breathing.

FIG. 4A depicts a bowl after a child has consumed the foodstuff(s)retained within the bowl (here, cereal and milk). After the child hasfinished consuming some or all of the foodstuff(s), the parent or thechild may once again scan the bowl by initiating a live view that iscaptured by the optical sensor (e.g., a camera) of a computing device(e.g., a mobile phone). The computing device may be the same computingdevice used to prompt creation of the augmented reality feature or adifferent computing device.

An application executing on the computing device may detect that some orall of a given foodstuff is no longer present, which is indicative ofconsumption by the child. For example, the application may comparefeature(s) recognized within individual video frames of a current livefeed to feature(s) recognized within individual video frames of aprevious live feed.

As the child consumes the foodstuff, augmented reality feature(s) hiddenwithin or beneath the foodstuff become increasingly visible. That is,the application may present a “payoff” upon determining that thefoodstuff has been consumed by the child. Such a technique incentivizesthe child to continue eating certain foodstuff(s) so that thecorresponding augmented reality feature(s) can be fully seen.

In some embodiments, the “payoff” simply includes presenting an entireaugmented reality feature for viewing by the child. FIGS. 4B-C, forexample, depict examples of augmented reality features that could beshown upon determining the child has consumed substantially all of thecorresponding foodstuff(s). In other embodiments, the “payoff” is adigital collectible corresponding to the augmented reality feature.

FIG. 5 illustrates how augmented reality features (also referred to as“digital collectibles”) acquired by a child may be part of a collection.Here, for example, the augmented reality feature of FIG. 4B is one ofsix augmented reality features that can be collected by the child. Thechild may collect the various augmented reality features of thecollection by consuming different types of foodstuffs, consuming thesame type of foodstuffs multiple times, etc.

Collections may be created by an individual (e.g., a parent or someother person, such as a dietician or physician) or an entity responsiblefor supporting the application. For example, a parent could specify thatsix different augmented reality features should be associated with sixdifferent types of foodstuff that a child typically does not consume. Asanother example, an entity could specify that six different augmentedreality features should be associated with type(s) of foodstuff that arenecessary to maintain a healthy diet.

The augmented reality features may also be used to facilitate differentforms of social gaming that require social interaction between multipleplayers (e.g., children). For example, children may be able to care foraugmented reality features, trade augmented reality features, battleaugmented reality features, etc. Accordingly, the augmented realityfeatures described herein could be used in the context of:

-   -   Card games;    -   Social networking games that are integrated into a social        network (e.g., Facebook®);    -   Board games in which augmented reality features permit usage of        certain components/pieces; and    -   Video games in which augmented reality features permit usage of        certain characters or items.

Such conduct may be permitted regardless of the type of augmentedreality feature generated by the application. For example, childrencould battle augmented reality features that take the form ofdigitally-animated creatures or trading cards.

FIG. 6 depicts an example of a network environment 600 that includes amobile phone 602 having an application configured to present augmentedreality features that are embedded within or disposed beneathfoodstuff(s) and a network-accessible server system 604 responsible forsupporting the application. Generally, the network environment 600 willonly include the mobile phone 602 and the network-accessible serversystem 604. However, in some embodiments the network environment 600also includes another computing device 606 (here, a head-mounted device)in addition to, or instead of, the mobile phone 602.

While many of the embodiments described herein involve mobile phones,those skilled in the art will recognize that such embodiments have beenselected for the purpose of illustration only. The technology could beused in combination with any computing device that is able to presentaugmented reality content, including personal computers, tabletcomputers, personal digital assistants (PDAs), game consoles (e.g., SonyPlayStation® or Microsoft Xbox®), music players (e.g., Apple iPodTouch®), wearable electronic devices (e.g., watches or fitness bands),network-connected (“smart”) devices (e.g., televisions and homeassistant devices), virtual/augmented reality systems (e.g.,head-mounted displays such as Oculus Rift® and Microsoft Hololens®), orother electronic devices.

For example, a child could wear a head-mounted display that allows thechild to have a live direct/indirect view of a physical, real-worldenvironment whose elements are augmented by computer-generated sensoryinput (e.g., one or more augmented reality features). The augmentedreality feature(s) may be created responsive to instructions generatedby the head-mounted display or another computing device to which thehead-mounted display is communicatively coupled. For example, anapplication residing on a mobile phone 602 associated with a parent maygenerate instructions for creating the augmented reality feature(s) andtransmit those instructions to the head-mounted display. As anotherexample, a network-connected server system 604 may generate instructionsfor creating the augmented reality feature(s) and transmit thoseinstructions to the mobile phone 602 or the head-mounted display.

Accordingly, the mobile phone 602, the network-connected server system604, and/or the other computing device 606 can be connected via one ormore computer networks 608 a-c, which may include the Internet, localarea networks (LANs), wide-area networks (WANs), metropolitan areanetworks (MANs), cellular networks (e.g., LTE, 3G, 4G), etc.Additionally or alternatively, the mobile phone 602, thenetwork-connected server system 604, and/or the other computing device606 may communicate with one another over a short-range communicationprotocol, such as Bluetooth®, Near Field Communication (NFC),radio-frequency identification (RFID), etc.

Generally, an application executing on the mobile phone 602 isresponsible for generating and presenting augmented reality features toan individual (e.g., a parent or a child). The network-connected serversystem 604 and/or the other computing device 606 can be coupled to themobile phone 602 via a wired channel or a wireless channel. In someembodiments the network-accessible server system 604 is responsible fordelivering augmented reality assets (e.g., markers, textures, animationinstructions, audio files) to the mobile phone 602, while in otherembodiments the mobile phone 602 stores some or all of the augmentedreality assets locally (e.g., within a memory). Therefore, in someinstances the mobile phone 602 may execute the techniques describedherein without needing to be communicatively coupled to any network(s),other computing devices, etc.

FIG. 7 depicts an application 700 that allows augmented reality featuresto be created for presentation by a computing device. In someembodiments the application 700 is running on the computing device(e.g., a mobile phone), while in other embodiments the application 700is running on another computing device (e.g., a network-connected serversystem) that is communicatively coupled to the computing device.

The application 700 can include a graphical user interface (GUI) module702 that generates an interface 710 through which an individual 712(e.g., a parent, dietician, or physician) can provide instructions forcreating augmented reality feature(s). The interface 710 may beaccessible via a web browser, desktop software program, mobile softwareapplication, or over-the-top (OTT) application.

More specifically, the application 700 can determine whether to createaugmented reality features based on instructions provided by theindividual 712 through the interface 710. For example, the individual712 may scan one or more foodstuffs by initiating a live view that iscaptured by a camera of the computing device, and then provideinstructions at the interface 710 for generating the augmented realityfeature(s). The instructions may specify a particular foodstuff, anaugmented reality feature type, an augmented reality feature color, anaugmented reality feature animation, etc. The instructions can be storedin one or more storage modules 704 that are accessible to theapplication 700. Thus, feature creation may be completed via theinterface 710 that is created and supported by the application 700.

In some embodiments, a processing module 706 of the application 700 isconfigured to automatically identify recognizable foodstuff features onbehalf of the individual 712. The processing module 706 may apply imageprocessing algorithms to identify certain shapes and/or colors that areindicative of certain foodstuffs (e.g., green spheres for peas, orangecylinders for carrots, white liquid for milk). More specifically, theprocessing module 706 may perform image segmentation on individualframes of the live video feed captured by the camera of the computingdevice to isolate regions and objects of interest.

In other embodiments, the application 700 identifies foodstuff(s) basedon user input received from the individual 712 who indicates whichfoodstuff(s) should be associated with an augmented reality feature. Forexample, the individual 712 may be able to simply select a certainfoodstuff shown on the interface 710.

A feature creation module 708 can then create an augmented realityfeature that is at least partially hidden beneath a foodstuff. Forexample, portions of a digitally-animated creature may be visible whenthe foodstuff is viewed through a computing device, as shown in FIG. 3.The augmented reality features can take many different forms, includingcreatures (e.g., humanoid characters and non-humanoid characters, suchas dragons and dinosaurs) and items (e.g., treasure chests, tradingcards, jewels, coins).

The feature creation module 708 may also generate instructions thatcause the augmented reality feature to perform animation(s). Forexample, different appendages (e.g., arms, legs, tails, head) may emergefrom a foodstuff over time. As another example, the foodstuff itself mayappear as though it is animated (e.g., the surface of a foodstuff mayinclude ripples or bubbles).

FIG. 8 depicts a flow diagram of a process 800 for incentivizing theconsumption of foodstuff through the use of augmented reality features.Initially, an application is launched on a mobile phone that includes acamera (step 801). Although the embodiment described herein pertains toa mobile phone having a camera, those skilled in the art will recognizethat similar steps can be performed using other types of computingdevices and optical sensors.

The application can then cause a communication link to be establishedbetween the mobile phone and a network-connected server (step 802). Thenetwork-connected server (or some other computing device) may be part ofa network-connected server system that supports various back-endfunctionalities of the application.

An individual can then initiate a live view of one or more foodstuffsthat is captured by the camera (step 803). The application identifiessome or all of the foodstuff(s) included in the live view (step 804).For example, in some embodiments the application may perform imageprocessing in order to automatically identify shapes, colors, or somecombination thereof that are indicative of certain foodstuffs, while inother embodiments the application may receive user input indicative of aselection of a foodstuff.

The application can then generate an augmented reality feature that isat least partially hidden beneath a particular foodstuff (step 805). Forexample, portions of a digitally-animated creature may be visible whenthe foodstuff is viewed through a mobile phone. The application may alsoanimate a portion of the augmented reality feature that is exposed fromthe particular foodstuff (step 806). For example, a tail protruding froma foodstuff may slowly move back and forth. Additionally oralternatively, the application may animate the particular foodstuff. Forexample, the application may cause ripples to appear in the foodstuffproximate to the tail and/or bubbles to appear in the foodstuffproximate to the creature's head.

After creating the augmented reality feature, the application canmonitor the live view that is captured by the camera (step 807). Suchaction may occur during a single viewing session or across multipleviewing sessions. For example, a first individual (e.g., a parent) mayinitiate a first viewing session during which the augmented realityfeature is created, and then a second individual (e.g., a child) mayinitiate a second viewing session during which the augmented realityfeature can be observed. The second viewing session typically occursduring or after consumption of the particular foodstuff.

In some instances, the application will determine that the particularfoodstuff has been partially or entirely consumed (step 808). Generally,the application arrives at such a determination by performing imageprocessing on individual frames of the live view. Note, however, theprocess 800 could also be carried out on static images captured by thecamera rather than a live view. For example, the application may comparefeature(s) recognized within individual video frames of a current livefeed to feature(s) recognized within individual video frames of aprevious live feed.

The application then displays one or more additional portions of theaugmented reality feature based on the amount of the particularfoodstuff that has been consumed (step 809). Said another way, theapplication can expose additional portion(s) of the augmented realityfeature as an individual (e.g., a child) consumes the particularfoodstuff.

In some embodiments, the application is configured to generate metadataspecifying that the additional portion(s) of the augmented realityfeature have been exposed due to consumption of the particular foodstuff(step 810). Moreover, the metadata may be transmitted to thenetwork-connected server responsible for supporting the application(step 811). Reception of the metadata may prompt the network-connectedserver to create a database record that specifies the augmented realityfeature should be made accessible to an individual (e.g., a child)through the application or another application. Accordingly, theaugmented reality feature may be one of several different augmentedreality features that can be collected by a child by consuming differenttypes of foodstuffs, consuming the same type of foodstuffs multipletimes, etc.

FIG. 9 depicts a flow diagram of a process 900 for creating augmentedreality features that can be used to incentivize the consumption offoodstuffs. Steps 901-903 are largely identical to steps 801-803 of FIG.8.

In order to facilitate the creation of augmented reality features, theapplication may be configured to automatically identify some or all ofthe foodstuff(s) visible through the live feed (step 904). For example,the application may apply image processing algorithms to identifycertain shapes and/or colors that are indicative of certain foodstuffs(e.g., green spheres for peas, orange cylinders for carrots, whiteliquid for milk). More specifically, the application may perform imagesegmentation (e.g., thresholding methods such as Otsu's method, orcolor-based segmentation such as K-means clustering) on individualframes of the live video feed to isolate regions and objects ofinterest.

While the application may be configured to automatically identifyrecognizable features, an individual (e.g., a parent) will typicallyneed to indicate which foodstuff(s) on a plate should be associated withan augmented reality feature. Accordingly, the application may receiveuser input indicative of a selection of a particular foodstuff (step905). Generally, user input is created when the individual taps thescreen of a mobile phone in the location where the particular foodstuffis being shown. However, other input devices may also be used.

The application may also receive user input indicative of aspecification of a characteristic of the augmented reality feature to begenerated (step 906). For example, the individual may specify a certainfeature type (e.g., a particular creature such as a dragon or dinosaur,or a particular item such as a trading card), a certain color, a certainanimation, etc. The application can then generate the augmented realityfeature in accordance with the user input(s) (step 907).

Unless contrary to physical possibility, it is envisioned that the stepsdescribed above may be performed in various sequences and combinations.For example, in some embodiments a parent may select the type ofaugmented reality feature to be displayed prior to selecting thefoodstuff in which the augmented reality feature is at least partiallyburied.

Additional steps could also be included in some embodiments. In someembodiments. the application may continually monitor the live feed inorder to determine which portion(s) of the augmented reality featureshould be digitally exposed at any given point in time. Moreover, theportion(s) may change over time as a child consumes/moves thecorresponding foodstuff. For example, different portion(s) of theaugmented reality feature may be visible if the child shifts a mound ofpeas around, but the entirety of the augmented reality feature may notbe visible until most or all of the peas have been consumed.

In fact, a parent could specify a threshold level of consumption that isnecessary before the augmented reality feature can be “acquired” by achild. For example, the parent may specify that the child must consume50%, 75%, or 95% of the corresponding foodstuff before the augmentedreality feature is made available through other channels (e.g., socialmedia networks, digital collections, or video games).

Processing System

FIG. 10 is a block diagram illustrating an example of a processingsystem 1000 in which at least some operations described herein can beimplemented. The processing system may include one or more centralprocessing units (“processors”) 1002, main memory 1006, non-volatilememory 1010, network adapter 1012 (e.g., network interfaces), videodisplay 1018, input/output devices 1020, control device 1022 (e.g.,keyboard and pointing devices), drive unit 1024 including a storagemedium 1026, and signal generation device 1030 that are communicativelyconnected to a bus 1016.

The bus 1016 is illustrated as an abstraction that represents one ormore physical buses and/or point-to-point connections that are connectedby appropriate bridges, adapters, or controllers. Therefore, the bus1016 can include a system bus, a Peripheral Component Interconnect (PCI)bus or PCI-Express bus, a HyperTransport or industry standardarchitecture (ISA) bus, a small computer system interface (SCSI) bus, auniversal serial bus (USB), IC (I2C) bus, or an Institute of Electricaland Electronics Engineers (IEEE) standard 1394 bus (also referred to as“Firewire”).

In some embodiments the processing system 1000 operates as part of amobile phone that executes an application configured to generateaugmented reality features, while in other embodiments the processingsystem 1000 is connected (wired or wirelessly) to the mobile phone. In anetworked deployment, the processing system 1000 may operate in thecapacity of a server or a client machine in a client-server networkenvironment, or as a peer machine in a peer-to-peer network environment.The processing system 1000 may be a server, a personal computer (PC), atablet computer, a laptop computer, a personal digital assistant (PDA),a mobile phone, a processor, a telephone, a web appliance, a networkrouter, a switch, a bridge, a console, a gaming device, a music player,or any machine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by the processing system1000.

While the main memory 1006, non-volatile memory 1010, and storage medium1026 (also called a “machine-readable medium”) are shown to be a singlemedium, the term “machine-readable medium” and “storage medium” shouldbe taken to include a single medium or multiple media (e.g., acentralized or distributed database and/or associated caches andservers) that store one or more sets of instructions 1028. The term“machine-readable medium” and “storage medium” shall also be taken toinclude any medium that is capable of storing, encoding, or carrying aset of instructions for execution by the processing system 1000.

In general, the routines executed to implement the embodiments of thedisclosure may be implemented as part of an operating system or aspecific application, component, program, object, module or sequence ofinstructions referred to as “computer programs.” The computer programstypically comprise one or more instructions (e.g., instructions 1004,1008, 1028) set at various times in various memory and storage devicesin a computing device, and that, when read and executed by the one ormore processors 1002, cause the processing system 1000 to performoperations to execute elements involving the various aspects of thetechnology.

Moreover, while embodiments have been described in the context of fullyfunctioning computing devices, those skilled in the art will appreciatethat the various embodiments are capable of being distributed as aprogram product in a variety of forms. The disclosure applies regardlessof the particular type of machine or computer-readable media used toactually effect the distribution.

Further examples of machine-readable storage media, machine-readablemedia, or computer-readable media include, but are not limited to,recordable-type media such as volatile and non-volatile memory devices1010, floppy and other removable disks, hard disk drives, optical disks(e.g., Compact Disk Read-Only Memory (CD ROMS), Digital Versatile Disks(DVDs)), and transmission-type media such as digital and analogcommunication links.

The network adapter 1012 enables the processing system 1000 to mediatedata in a network 1014 with an entity that is external to the processingsystem 1000 through any communication protocol supported by theprocessing system 1000 and the external entity. The network adapter 1012can include one or more of a network adaptor card, a wireless networkinterface card, a router, an access point, a wireless router, a switch,a multilayer switch, a protocol converter, a gateway, a bridge, bridgerouter, a hub, a digital media receiver, and/or a repeater.

The network adapter 1012 can include a firewall that governs and/ormanages permission to access/proxy data in a computer network, andtracks varying levels of trust between different machines and/orapplications. The firewall can be any number of modules having anycombination of hardware and/or software components able to enforce apredetermined set of access rights between a particular set of machinesand applications, machines and machines, and/or applications andapplications (e.g., to regulate the flow of traffic and resource sharingbetween these entities). The firewall may additionally manage and/orhave access to an access control list that details permissions includingthe access and operation rights of an object by an individual, amachine, and/or an application, and the circumstances under which thepermission rights stand.

The techniques introduced here can be implemented by programmablecircuitry (e.g., one or more microprocessors), software and/or firmware,special-purpose hardwired (i.e., non-programmable) circuitry, or acombination of such forms. Special-purpose circuitry can be in the formof, for example, one or more application-specific integrated circuits(ASICs), programmable logic devices (PLDs), field-programmable gatearrays (FPGAs), etc.

Remarks

The foregoing description of various embodiments of the claimed subjectmatter has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit the claimedsubject matter to the precise forms disclosed. Many modifications andvariations will be apparent to one skilled in the art. Embodiments werechosen and described in order to best describe the principles of theinvention and its practical applications, thereby enabling othersskilled in the relevant art to understand the claimed subject matter,the various embodiments, and the various modifications that are suitedto the particular uses contemplated.

Although the above Detailed Description describes certain embodimentsand the best mode contemplated, no matter how detailed the above appearsin text, the embodiments can be practiced in many ways. Details of thetechnology may vary considerably in its implementation details whilestill being encompassed by the specification. As noted above, particularterminology used when describing certain features or aspects of variousembodiments should not be taken to imply that the terminology is beingredefined herein to be restricted to any specific characteristics,features, or aspects of the invention with which that terminology isassociated. In general, the terms used in the following claims shouldnot be construed to limit the invention to the specific embodimentsdisclosed in the specification, unless those terms are explicitlydefined herein. Accordingly, the actual scope of the inventionencompasses not only the disclosed embodiments, but also all equivalentways of practicing or implementing the embodiments covered by theclaims.

The language used in the specification has been principally selected forreadability and instructional purposes, and it may not have beenselected to delineate or circumscribe the inventive subject matter. Itis therefore intended that the scope of the invention not be limited bythis Detailed Description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of variousembodiments is intended to be illustrative, but not limiting, of thescope of the technology.

What is claimed is:
 1. A non-transitory computer-readable medium withinstructions stored thereon that, when executed by a processor, causethe processor to perform operations comprising: causing an opticalsensor of a computing device to capture a first image of a physicalobject indicative of a foodstuff; establishing a characteristic of thephysical object by examining the first image; and generating anaugmented reality feature that is at least partially hidden beneath thefoodstuff when viewed using the computing device.
 2. The non-transitorycomputer-readable medium of claim 1, wherein the first image is one of aseries of images that is representative of a video.
 3. Thenon-transitory computer-readable medium of claim 1, wherein the firstimage is one of a series of images captured by the optical sensor duringa live view of the physical object.
 4. The non-transitorycomputer-readable medium of claim 1, wherein the physical object is apackage in which the foodstuff resides, a piece of tableware, or thefoodstuff.
 5. The non-transitory computer-readable medium of claim 1,wherein the characteristic is a surface contour, a shape, a color, orany combination thereof.
 6. The non-transitory computer-readable mediumof claim 1, wherein the operations further comprise: animating a portionof the augmented reality feature that is exposed from the foodstuff. 7.The non-transitory computer-readable medium of claim 1, wherein theoperations further comprise: animating a surface of the foodstuff toindicate that the augmented reality feature is entirely hidden withinthe foodstuff.
 8. The non-transitory computer-readable medium of claim1, wherein the operations further comprise: causing the optical sensorof the computing device to capture a second image of the physical objectindicative of the foodstuff; determining, based on a comparison of thefirst and second images, that the foodstuff has been at least partiallyconsumed; and displaying a portion of the augmented reality feature thatwas previously hidden beneath the foodstuff, wherein the portion isbased on an amount of the foodstuff that has been consumed.
 9. Thenon-transitory computer-readable medium of claim 8, wherein theoperations further comprise: generating metadata specifying that theportion of the augmented reality feature has been exposed; andtransmitting the metadata to a network-connected server systemconfigured to maintain a list of consumed foodstuffs, exposed augmentedreality features, or any combination thereof.
 10. The non-transitorycomputer-readable medium of claim 9, wherein reception of the metadataprompts the network-connected server system to create a record thatspecifies the augmented reality feature should be made accessiblethrough a software program executing on the computing device.
 11. Anetwork-connected computer server comprising: a memory that includesinstructions for supporting a software program designed to incentivizethe consumption of foodstuffs, wherein the instructions, when executedby a processor, cause the processor to: identify an instance of thesoftware program that is executing on a computing device; receive, fromthe instance of the software program, metadata that specifies a portionof an augmented reality feature that was previously hidden beneath afoodstuff has been exposed due to consumption of the foodstuff; create arecord that specifies the augmented reality feature should be madeaccessible through the instance of the software program; and enable anindividual to interact with the augmented reality feature through theinstance of the software program.
 12. The network-connected computerserver of claim 11, wherein the augmented reality feature is a digitalcreature or a digital item.
 13. The network-connected computer server ofclaim 11, wherein the computing device is one of multiple computingdevices that are communicatively coupled to the network-connectedcomputer server, and wherein each computing device of the multiplecomputing devices executes a separate instance of the software program.14. A computer-implemented method comprising: initiating a softwareprogram on a computing device that includes an optical sensor; causingthe optical sensor to capture a first image of a foodstuff; andgenerating an augmented reality feature that is at least partiallyhidden beneath the foodstuff when viewed using the computing device. 15.The computer-implemented method of claim 14, further comprising:identifying the foodstuff by applying an image processing algorithmdesigned to identify a shape or a color indicative of a given foodstuffto the first image.
 16. The computer-implemented method of claim 14,further comprising: identifying the foodstuff by recognizing a brandedcode that conveys information about a given foodstuff.
 17. Thecomputer-implemented method of claim 16, wherein the branded code isprinted on a package in which the foodstuff resides, a loose itemsecured within the package, or a receipt for a purchase of thefoodstuff.
 18. The computer-implemented method of claim 16, wherein thebranded code includes a machine-readable element, a human-readableelement, a structural element, or any combination thereof.
 19. Thecomputer-implemented method of claim 16, wherein the branded codeincludes a bar code or a Quick Response (QR) code that is scannable bythe software program.
 20. The computer-implemented method of claim 14,further comprising: causing the optical sensor to capture a second imageof the foodstuff; determining, based on a comparison of the first andsecond images, that the foodstuff has been at least partially consumed;and displaying a portion of the augmented reality feature that waspreviously hidden beneath the foodstuff, wherein the portion is based onan amount of the foodstuff that has been consumed.